Fostering Safe AI Use: Jay Anders from Medicomp in a Riveting Conversation with PharmaShots

Shots:

Nearly half of FDA-approved AI medical devices lack testing on real patient data, raising significant concerns about patient safety.

PharmaShots welcomes Jay Anders, CMO of Medicomp Systems, to discuss AI implementation trends in healthcare and strategies for ethical AI adoption by healthcare professionals.

Jay emphasizes the importance of clinical validations to ensure the safe and effective use of AI in healthcare.

Saurabh: What are the key implications of utilizing synthetic data to train AI medical devices in terms of patient safety and treatment outcomes?

Jay: The use of synthetic data in AI medical devices raises serious concerns for patient safety and treatment efficacy. According to recent research published in Nature Medicine, nearly half of FDA-approved AI medical devices have not been trained on real patient data, which creates a significant gap between laboratory performance and real-world clinical effectiveness.

While synthetic data offers a potential solution to data access challenges, it often fails to capture the true complexity and nuance of actual patient cases. This disconnect could lead to inaccurate diagnoses, inappropriate treatment recommendations, and potentially harmful outcomes. The use of synthetic data, while expedient for development, may not adequately prepare AI systems for the variability and complexity encountered in real clinical settings. Additionally, as revealed in these findings, FDA authorization doesn't necessarily indicate proper clinical evaluation using real patient data, suggesting that current regulatory approval processes may not sufficiently guarantee patient safety.

Saurabh: How can transparency in AI development increase patient trust and safety in healthcare applications?

Jay: Transparency in AI development plays a crucial role in building trust and ensuring patient safety in healthcare settings. Transparent AI allows healthcare providers to understand the decision-making logic behind AI recommendations, enabling them to quickly identify errors and incorrect suggestions before they impact patient care. This visibility is particularly important in clinical settings where mistakes can have serious consequences. Transparency also helps eliminate bias and ensure equitable care by allowing clinicians to verify that AI technologies are trained on diverse datasets that accurately reflect their patient populations. It enables ongoing monitoring to maintain accuracy and fairness over time.

Transparency requirements should include clear documentation of training data, ethical considerations, and deployment limitations. The current transparency scores of authorized medical AI products in Europe, ranging from just 6.4% to 60.9% with a median of 29.1%, indicate significant room for improvement. By increasing transparency, healthcare organizations can better validate AI systems' reliability, ensure compliance with regulations, and ultimately foster greater trust among both healthcare providers and patients.

Saurabh: What are your thoughts on the present level of AI usage in healthcare? What concerns do you see if this trend continues without adequate oversight?

Jay: The current trajectory of AI implementation in healthcare presents several concerning issues. The emergence of "Instafraud" –– where AI is used to generate false or exaggerated medical documentation - demonstrates how these technologies can be misused for financial gain at the expense of patient care. There's also growing evidence of inadequate validation practices, with many AI systems being deployed without proper testing on real patient data.

Without adequate oversight, these issues could worsen, potentially leading to:

Increased healthcare disparities due to algorithmic bias

Patient harm from inaccurate or dangerous predictions

Privacy and security breaches of sensitive medical data

Erosion of trust in healthcare systems

Financial waste and fraud through AI-enabled upcoding

Compromised clinical decision-making due to over-reliance on improperly validated AI systems

Rushing AI deployment without proper safeguards could undermine the potential benefits these technologies offer while introducing new risks to patient care and safety.

Saurabh: What strategies may healthcare professionals employ to ensure responsible and ethical AI implementation in clinical practice?

Jay: Healthcare professionals can implement several strategies to ensure responsible and ethical AI use in clinical practice. First, they should prioritize real-time auditing of AI systems to flag suspicious patterns and enable immediate human review. Enhancing interoperability between healthcare providers and insurers to reduce conflicting diagnoses and improve data sharing is essential.

Additionally, healthcare organizations should:

Invest in ongoing training for healthcare professionals on proper coding practices and ethical AI use.

Implement standardized protocols for AI usage with clear guidelines on human oversight.

Establish point-of-care solutions that allow clinical providers to validate AI outputs in real time.

Maintain robust documentation practices and transparency in AI-assisted decision-making.

Develop clear protocols for when to trust or question AI recommendations.

Ensure diverse representation in both development teams and training data.

Regularly assess AI systems for bias and accuracy.

These strategies should be implemented within a framework that maintains human judgment as the final authority in clinical decisions.

Saurabh: How crucial is high-quality, diverse patient data to the effectiveness of AI medical devices?

Jay: High-quality, diverse patient data is fundamental to the effectiveness of AI medical devices. The lack of real patient data in training sets is a critical issue that undermines the reliability and safety of AI systems in healthcare. Quality data directly impacts an AI system's ability to make accurate predictions and recommendations across different patient populations.

Research shows that many current AI systems are trained on limited or synthetic datasets, which may not capture the full spectrum of patient presentations and conditions. This limitation can lead to biased outcomes and reduced effectiveness when dealing with diverse patient populations. Without representative data from various demographic groups, AI systems may perpetuate or even exacerbate existing healthcare disparities. Furthermore, quality data is essential for proper validation and testing of AI systems, ensuring they perform as intended across different clinical settings and patient populations.

Saurabh: What are the minimal requirements you believe should be established for AI validation in healthcare to ensure that these technologies are safe and useful to patients?

Jay: Several crucial minimal requirements should be established for AI validation in healthcare:

First, clinical validation must include testing with real patient data rather than synthetic data alone. This validation should follow a hierarchy of evidence, with retrospective validation as a minimum, prospective validation as preferred, and randomized controlled trials as the gold standard. Validation should occur in real-world clinical settings with diverse patient populations.

Second, transparency requirements must include:

Clear documentation of development and training processes

Detailed information about data sources and quality

Explicit statements of limitations and intended use cases

Regular monitoring and reporting of performance metrics

Public disclosure of validation results and safety data

Third, risk management practices should be mandatory, including:

Regular safety monitoring and updates

Clear protocols for identifying and addressing errors

Established procedures for human oversight

Regular audits by independent third parties

Comprehensive documentation of any adverse events or failures

These requirements should be legally mandated rather than voluntary to ensure consistent implementation across the healthcare industry.

Image Source: Canva

About the Author:

Jay Anders, MD

Jay Anders, MD, serves as the Chief Medical Officer of Medicomp Systems, where he plays a pivotal role in product development and acts as a key liaison to the healthcare community. He is a passionate advocate for leveraging technology to empower clinicians rather than impede them.

In his role, Dr. Anders leads Medicomp’s knowledge base team and clinical advisory board, ensuring that product development aligns with user needs to enhance usability. He also hosts the award-winning HealthcareNOW Radio podcast, Tell Me Where IT Hurts, where he discusses critical topics such as physician burnout, EHR usability, healthcare interoperability, and the impact of technology on healthcare with industry experts.

Before joining Medicomp, Dr. Anders served as the Chief Medical Officer at McKesson Business Performance Services, where he oversaw the development of clinical information systems and spearheaded the integration of Medicomp’s Quippe Physician Documentation into an EHR.

Dr. Anders began his medical career in a large multi-specialty practice, where he confronted firsthand the inefficiencies of health IT and the challenges of clinician burnout. After 16 years in internal medicine - as a physician, department chair, clinic president, and medical director - he transitioned from private practice to focus on using technology to improve patient care and address physician burnout.